perf(loki.source.file): Reduce allocations during file reading [backport]#5416
Merged
kalleep merged 1 commit intorelease/v1.13from Feb 2, 2026
Merged
perf(loki.source.file): Reduce allocations during file reading [backport]#5416kalleep merged 1 commit intorelease/v1.13from
kalleep merged 1 commit intorelease/v1.13from
Conversation
### Pull Request Details There is a lot of allocations being performed while reading files. I can clearly see this being a hot spot in our internal clusters. Two of the biggest issues are: 1. We allocate a new pending buffer as soon as we have completed a line 2. We use ReadBytes api causing new allocation for every call that we just move to pending Instead we can use `ReadSlice`, this api will not allocate but return a slice pointing to the internal buffer while still advancing read position. We copy it into pending so this is fine and we remove the intermediate allocation done by `ReadBytes`. We can also reset length of `pending` and this buffer. This is fine because we never read past newline boundary. ### Issue(s) fixed by this Pull Request <!-- Fixes #issue_id --> ### Notes to the Reviewer <!-- Add any relevant notes for the reviewers and testers of this PR. --> ### PR Checklist <!-- Remove items that do not apply. For completed items, change [ ] to [x]. --> - [ ] Documentation added - [x] Tests updated - [ ] Config converters updated (cherry picked from commit 899467a)
kalleep
approved these changes
Feb 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #5405
This PR backports #5405 to release/v1.13.
Original PR Author
@kalleep
Description
Pull Request Details
There is a lot of allocations being performed while reading files. I can clearly see this being a hot spot in our internal clusters.
Two of the biggest issues are:
Instead we can use
ReadSlice, this api will not allocate but return a slice pointing to the internal buffer while still advancing read position. We copy it into pending so this is fine and we remove the intermediate allocation done byReadBytes.We can also reset length of
pendingand this buffer. This is fine because we never read past newline boundary.Issue(s) fixed by this Pull Request
Notes to the Reviewer
PR Checklist
This backport was created automatically.